Multi-directional remote control system and method

ABSTRACT

A multi-directional remote control system and method is adapted for use with an entertainment system of a type including a display such as a monitor or TV and having display functions employing a mouse type control. The remote controller may be conveniently held in one hand of a user and still provides full mouse type functionality. The remote control system and method images the screen of the TV or other display to detect relative motion between the controller and screen. This position information is transmitted to the entertainment system for control of a cursor or other GUI interface.

RELATED APPLICATION INFORMATION

The present application claims priority to provisional application Ser.No. 60/622,143 filed Oct. 25, 2004, the disclosure of which isincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to remote control systems for controllingentertainment systems, such as televisions, multimedia systems, Internetaccess systems and browsers, and related methods.

2. Description of the Prior Art and Related Information

A need has arisen for providing multi-directional mouse type controlcapabilities in the living room along with the ability to control theconventional entertainment devices typically present in the living room.For example, combined PC and TV systems have been introduced whichintegrate the capabilities of the personal computer with the television.One such system is described in U.S. Pat. No. 5,675,390. Also, set topInternet access devices have been introduced which integrate Internetaccess capabilities with conventional televisions. The ability toprovide full control of a PC or an Internet browser typically requiresthe use of a keyboard and a multi-directional controller such as amouse. A conventional remote control is therefore inadequate for controlof such combined entertainment systems. Also, the advent of digitalvideo recorders (DVRs), wireless networking systems for video, audio andpicture transfer to TVs, and other digital devices linked to the TV hasintroduced many more functions to TV control, including complex displaymenus, introducing a need for a better remote control interface.

Wireless keyboards are one addition to the conventional remote controlin the living room that have been introduced to allow the user of acombined PC and TV system or the user of a TV Internet access device toprovide convenient text input, for example for creating emails orsearching. However, convenient control of PC type functions alsorequires an ability to interface with a Graphical User Interface (GUI).To address this need wireless keyboards may include anup-down-left-right control to move around in a limited GUI interface.This type of up-down-left-right control is also typically added toconventional remotes and used to navigate a cable TV menu or digital TVperipheral device menu, such as a DVR. This type of up-down-left-rightcontrol is more restricted and clumsy to use than a mouse typecontroller and limits the flexibility of a GUI interface and the menulayout. Alternatively, wireless keyboards may include an integratedtrackball or other pointing device to provide mouse type control of thePC or Internet functions. These types of multi-directional controls areless natural and convenient to use than a separate mouse controller.Also, such systems require both hands to use making simple one handednavigation of a GUI TV interface impossible. A wireless mouse controlleris an option, however, a mouse requires a clean flat surface within easyreach and is not convenient for a living room setting. Some attemptshave been made to provide a mouse type controller suitable for livingroom use, for example, using gyroscopic motion detection, however suchcontrollers suffer from various problems such as cost, complexity andlack of naturalness of use. Furthermore, to provide all the desiredtypes of controls of a PC/TV entertainment system three separatewireless remote controls would be needed, a hand-held remote control, awireless keyboard and a freely movable mouse type control. This ofcourse introduces undesirable cost, a confusing number of controlfunctions, and clutter in the living room.

Accordingly, the addition of complex digital devices as well as PCand/or Internet access capabilities to the conventional TV basedentertainment system has introduced the problem of controlling suchsystems with a convenient yet full function remote control system.

SUMMARY

In a first aspect the present invention provides a method for remotecontrol of a media system including a display using a remote controlhaving a camera. The method comprises displaying an image on the displayusing the display, capturing image data of the displayed image anddisplay screen using the camera on the remote control, and determiningimage position information from the captured image data. The methodfurther comprises controlling at least a portion of the displayed imagein response to movement of the remote control using the detectedposition information.

In a preferred embodiment of the method for remote control of a mediasystem, displaying an image comprises displaying a graphical userinterface or video game. The image displayed by the display may includea distinctive boundary or marker and determining image positioninformation may comprise detecting the distinctive border or marker. Forexample, the displayed image may comprise externally provided images andthe distinctive boundary or marker is superimposed on the externallyprovided images. Determining image position information may alsocomprise detecting the display screen boundary. The graphical userinterface displayed may include a cursor and controlling at least aportion of the displayed image may comprise controlling the position ofthe displayed cursor. Alternatively, controlling at least a portion ofthe displayed image may simply comprise selectively highlightingportions of a graphical user interface or may comprise control of motionin a video game. Controlling at least a portion of the displayed imageusing the position information may also comprise wirelessly transmittingthe position information from the remote control to the display or adevice coupled to the display. The position information may preferablybe derived from offset information of a detected characteristic featureof the imager data from the camera optical axis.

According to another aspect the present invention provides a method forremote control of an entertainment system including a televisiondisplay. The method comprises displaying an image comprising a menu onthe television display in response to activation of a control input onthe remote, the menu including channel information, entertainment systemcontrol information, or internet access information. The method furthercomprises capturing image data of the television display including themenu using an imager in a freely movable remote control and detectingcharacteristic image features of the menu or the display screenboundary. The method further comprises providing multi-directionalcontrol of the display from changes in the detected characteristic imagefeatures due to movement of the remote control.

In one preferred embodiment of the method for remote control of a mediasystem the multi-directional control of the display may compriseproviding rapid scrolling through listed TV channels in response tomovement of the remote control. The characteristic image features of themenu may comprise a menu boundary or a marker displayed on the menu.Also, the characteristic image features of the menu may comprise acharacteristic color of the boundary or marker. The multi-directionalcontrol may comprise converting detected motion of the remote control toa sequence of up, down, left or right coded control pulses andtransmitting the sequence of pulses to the television display or devicecoupled to the display.

According to another aspect the present invention provides anentertainment system and remote control combination. The combinationcomprises a television display and a device integrated with, or separateand coupled to, the display for controlling display of images on thedisplay including externally provided internet information andtelevision images, the device including processing circuitry forproviding a distinctive boundary or marker on the displayed image. Thecombination also includes a freely movable remote control including anon board imager and optics adapted to capture image data including thedisplayed image, processing circuitry for detecting the distinctiveboundary or marker and deriving multi-directional position controlinformation, and transmission circuitry for transmitting the positioncontrol information to the device or display.

In a preferred embodiment of the entertainment system and remote controlcombination the distinctive boundary or marker may comprise a boundaryor marker with a characteristic brightness, color or shape. For example,the distinctive boundary may have a brightness allowing distinctivedetection of the edge of the display surrounding all or part of theimage.

According to another aspect the present invention provides a remotecontrol adapted for use with a television display providing a displayedimage on a television screen. The remote control comprises an imager,optics associated with the imager allowing a field of view including thetelevision screen and processing circuitry for detecting at least one ofthe screen boundary, a distinctive boundary displayed on the screen or adistinctive marker displayed on the screen and detecting movement of theremote control using the detected information. The remote controlfurther comprises transmission circuitry for transmittingmulti-dimensional control information derived from the detectedmovement.

In a preferred embodiment the remote control may further compriseprocessing circuitry providing universal remote control pulses inaddition to the multi-directional control information and/or processingcircuitry for providing universal remote control pulses derived from thedetected movement.

Further features and advantages of the present invention are set out inthe following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an improved entertainment system inaccordance with the present invention in a presently preferredembodiment.

FIG. 2 is a top view of the remote controller of the present inventionin a presently preferred embodiment.

FIG. 3 is a block schematic diagram illustrating control circuitry ofthe remote controller of the present invention.

FIG. 4 is a schematic diagram illustrating the image data captured bythe remote controller of the present invention including the displayscreen of the entertainment system of FIG. 1.

FIG. 5 is a schematic diagram illustrating the image data afterbackground processing, which image data corresponds to the desireddisplay screen image data, and derived relative position information.

FIG. 6 is a flow diagram illustrating the processing of image data bythe remote controller of the present invention.

FIG. 7 is a simplified schematic of the display control/input device ofthe system of FIG. 1.

FIG. 8 is a flow diagram illustrating the process flow of the displaycontrol/input device for converting the received position data to acursor or other GUI multi-directional control function.

FIGS. 9A and 9B illustrate display menus having a bright boundary and/ormarkings for use in image isolation.

FIG. 10 is a perspective view of an alternate embodiment of the remotecontroller of the present invention employing a folding configurationwith a text entry keyboard.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a remote control system and methodadapted for use with an entertainment system employing amulti-directional control function such as a GUI control interface. Anysuch multi-directional control capability is referred to herein, forshorthand purposes only, as a GUI interface. In FIG. 1 an improvedentertainment system in accordance with the present invention isillustrated in a perspective view in a presently preferred embodiment.Details of such systems beyond the novel control features describedherein are known and will not be described in detail herein. Forexample, a PC/TV system with internet access is one example of such anentertainment system and is disclosed in the above noted '390 patent,the disclosure of which is incorporated by reference in its entirety.

Referring to FIG. 1, the entertainment system 100 includes amulti-directional remote controller 110, a display 112, which forexample may be a TV monitor, a primary display control/input device 114and a secondary display control/input device 116. Primary displaycontrol/input device 114 and secondary display control/input device 116may comprise any of a variety of devices using a TV display for output.Primary control/input device 114 is adapted for a GUI interface controldisplayed on the display 112. For example, the primary input device 114may comprise a multi-media PC such as in the above noted '390 patent orother device adapted for utilizing a multi-directional control, such asa GUI interface. Other examples of primary input device 114 includedigital cable or satellite TV boxes, DVR systems, networked digitalmedia systems adapted for media transfer from a networked PC, internetsteaming media devices, digital video game players, etc. A variety ofpossible devices may therefore comprise primary input device 114.Furthermore the functionality of input device 114 may be incorporated inthe television system 112 and is simply illustrated as a separate devicefor illustration of one possible configuration. Secondary input device116 may also comprise any of a variety of known devices employed inentertainment systems and may include a DVR, cable TV box, or otherdigital or combined analog and digital interface device. Device 116 mayincorporate a GUI type interface or a more conventional interface for TVsystems adapted for, e.g. a push button LED remote control. Also, thefunctionality of device 116 may be incorporated along with device 114 orTV 112 and again the illustration of a separate input device is purelyfor illustration of a possible configuration and without limitation.Plural devices 114, 116 are shown to clarify that the control system ofthe present invention may control a conventional device as well as a GUIdevice, with an (optional) combined universal remote/multi-directionalcontrol capability in one embodiment of a controller 110 as describedbelow.

Remote controller 110 provides a multi-directional control capabilitywhich is schematically illustrated by control of cursor 118 displayed inthe monitor 112. It should be appreciated however that a variety ofdifferent multi-directional control interfaces may be employed otherthan a cursor such as in a typical mouse control of a PC. For examplethe multi-directional controller 110 may control highlighting andselection of different icons or other GUI interface layouts displayed onthe screen of TV monitor 112 by device 114 and/or device 116. Also, themulti-directional controller could simply enable rapid scrolling throughlarge channel lists such as in digital cable menus without the tediousup-down-left-right scrolling typically employed. As will be described inmore detail below, remote controller 110 employs a freely movablemulti-directional motion based control similar to a mouse control of aPC but without being limited to use on a flat surface.

Referring to FIG. 2, the remote controller 110 is illustrated in moredetail in a top view. As shown, the remote controller may have aconfiguration similar to a typical remote control employed in anentertainment system. Alternatively, the controller 110 may have a shapemore similar to a mouse type controller or other desirable ergonomicconfiguration adapted for use in one hand in a living room setting. Thetop surface of the controller housing 120 may include a number of firstremote control inputs indicated generally at 122. This first set ofcontrol inputs 122 may include conventional remote control functionstypically found in hand-held TV remote controls or universal remotecontrols adapted to control multiple entertainment devices such as TVs,VCRs, CD players, DVD players, etc. Therefore the first set of remotecontrol inputs 122 may include the volume up and down set of controls124, a channel up and down set of controls 126, a power button 128 and aset of numeric inputs 130. Also, a number of programmable or specialpurpose control buttons may be provided that are indicated generally asbuttons 132. The first set of controls 122 activate a first wirelesstransmitter 134 which may preferably be an LED or RF transmitterconfigured at one end of the housing 120. As further illustrated in FIG.2, the remote controller 110 preferably includes mouse type controlbuttons 136, 138 which comprise a second set of control inputs. Normallythe multi-directional control will not be needed and may be disabled forpower saving purposes. For this purpose a button 140 may be provided toactivate the second set of inputs and the multi-correctional controlcapability of the controller 110. Button 140 may at the same timetransmit a control signal to the control input device 114 to display asuitable menu adapted for multi-directional control on the displayscreen 112. Although one button 140 is shown several menu buttons may beprovided which enable display of the appropriate menu and at the sametime enable the multi-directional control capability. For example,activating a channel button would activate display of a channel listmenu and simultaneously enable the multi-directional control for rapidscrolling through the channels. Alternatively, the multi-directionalcontrol may only be active while button 140 is held depressed.Activation of the button 140 may also deactivate some or all of thefirst set of remote control inputs 122 so that these are notinadvertently activated while the user operates the mouse type controlbuttons 136 and 138. Alternatively a movable cover may be provided overthe first set of inputs 122 to cover these while the multi-directionalcontrol function is enabled; for example a sliding type cover may beprovided for this purpose or a hinged section with inputs 112 on theinner section or sections as shown by the dashed line. Alternatively, insome applications the remote control inputs 122 may not be needed; forexample in an application where the controller 110 is used inconjunction with a separate remote or with a keyboard having controlfunctions and the controller 110 is used solely as a multi-directionalinput device, some or all of the controls 122 may be dispensed with.Also, some or all of the functions of inputs 122 may be allocated to GUIcontrol on the screen. Also, the buttons 136, 138 may not be activatedby separate button(s) 140 but may be active all the time and operate asa conventional “action” or “select” button when operating in a non GUIcontrol mode and operate to select and/or provide mouse button typeselection when the multi-directional control mode is active. Althoughtwo buttons 136, 138 are shown similarly to a mouse a single button maybe employed (or more than two, or a scroll wheel may be added, e.g. forrapid channel navigation). The control signals from the control inputs136, 138 and the multi-directional control signals are provided to asecond transmitter 142 which may also preferably comprise either a LEDor RF type wireless transmitter. Alternatively, in some applications,e.g. video game control, a wired rather than wireless transmissionbetween the controller and device 114 may be preferred. As further shownin FIG. 2, a lens assembly 144 is provided at the front of the housingto allow the capture of image data including the display screen 112 forprovision to a digital camera (or imager) and image signal processingsystem described in more detail below.

The controller 110 may also provide various degrees of enhanced“universal control” GUI capability over various devices, such as device116 or TV 112. For example, most such devices will haveup-down-left-right controls and associated LED control pulses controlthe associated menu. If control of such devices using controller 110 isemployed, the detected motion of the controller 110 (described in detailbelow) can be converted to a high speed series of up, down, left orright LED pulses coded for the particular device being controlled. Inthis way more convenient navigation of conventional menus can beprovided by controller 110. Alternatively, device 114 may include an LED“blaster” or direct connection to other devices (e.g. via a USB port) toprovide such universal control.

Referring to FIG. 3, a block schematic diagram is illustrated showingthe circuitry of the remote controller. As shown in FIG. 3, thecontroller circuitry includes an imager 150 which receives lightcaptured by lens 144. Imager 150 may comprise a suitable commerciallyavailable digital imager, for example commercially available CMOSimagers providing relatively high-quality digital images are availableat relatively low cost and may be advantageously employed for imager150. The output of imager 150 will be image data corresponding to thepixels in the field of view of the imager 150, which field of view issuitably chosen by lens 144 to encompass the area in front of thecontroller including the display screen 112 shown in FIG. 1. The pixeldata output from imager 150 is provided to a signal processor 152 whichmay be a suitably programmed DSP programmed in a manner to provide theimage processing functions described in more detail below. The output ofthe DSP 152 will be data preferably corresponding to the position offset of the image of screen 112 shown in FIG. 1 from the image axis ofthe optics of remote controller 110. Alternatively, the data maycorrespond to changes in image position between frames. This positiondata is provided to microprocessor 154 which controls first transmitter134 to transmit the position data to the output control device 114 (or116) shown in FIG. 1. Microprocessor 154 will also receive inputs fromswitches 136 and 138 corresponding to the multi-directional controlbuttons shown in FIG. 2. These will also be provided to firsttransmitter 134 and provided to control/input device 114 (or 116) forcontrol of the GUI functions of the display 112. The microprocessor 154also receives inputs from activation of keys 122 (shown in FIG. 2)provided from key detect circuit 156. This key activation data isprovided by microprocessor 154 to second transmitter 142 and istransmitted to the appropriate input device 114, 116 or to TV 112. Twotransmitters 134 and 142 may be advantageously employed were the controlsignals from switches 122 provide a conventional LED type control signalwhich may be used for standard remote control functions in components inthe entertainment system illustrated in FIG. 1. Transmitter 134 in turnmay be optimized to transmit the position information and is preferablyinsensitive to the orientation of the control relative to the inputdevice (114) containing the receiving circuitry. Therefore, transmitter134 may be a sufficiently high bandwidth RF transmitter. Alternatively,however, transmitter 134 may also be an LED type transmitter. Also asingle transmitter may be employed for transmitting both types ofsignals under the control of microprocessor 154. Microprocessor 154 mayalso store codes for universal control operation. An (optional) receiver148 may also be provided, e.g. to receive a signal from device 114 thatthe multi-directional control menu has exited allowing controller 110 todisable the imager and DSP for power saving. Alternatively, as notedabove, the multi-directional control may only be active while a buttonis held depressed. Alternatively, a timer may put the imager and DSP ina sleep mode if a certain time has elapsed without use of a GUI controlbutton. Alternatively, the controller 110 may detect when the menu hasexited, for example, by detecting absence of a suitable boundary ormarker superimposed on the menu and disable the camera and DSP. Otherinformation may also be received from device 114, e.g. to customize thecontrol functions for different GUI interfaces. If device 114 has anetworked wireless interface, such as a WiFi interface, controller 110may also employ this protocol and be networked with device 114.Microprocessor 154 also receives as an input the control signal fromswitch 140 which, as noted above, may conveniently activate a menu orother interface signaling activation of the multi-direction controllerfunction and a GUI interface and optionally deactivate the controlinputs 122. Although the microprocessor 154 and DSP 152 are shown asseparate processors in FIG. 3, it will be appreciated that thefunctionality of these two processors may be combined into a singlemicroprocessor and the specific illustrated configuration of thecircuitry in FIG. 3 is purely one example for illustrative purposes.

Next, referring to FIGS. 4-6 the image processing implemented by DSP 152in FIG. 3 will be described in more detail. First of all, referring toFIG. 6 the first stage in the image processing is to capture a frame ofimage data as illustrated at 300. In FIG. 4 the image data captured byimager 150 is illustrated. As shown, the field of view 200 includesimage data (pixels) 202 corresponding to the display screen 112 shown inFIG. 1 as well as background image data 203. The image data 202 for thedisplay screen has several characteristics which distinguish it from thebackground and which allow it to be reliably detected by the imageprocessing software. These characteristics include the following: theimage data 202 from the display screen will be brighter than thebackground; the boundary of the image data 202 of the display screenwill have straight edges; the image 202 will have a rectangular shape;and the image 202 will have a substantial size in comparison to otherobjects in the total field of view 200. These characteristics may beemployed to eliminate the irrelevant background images and clearlydiscern the image 202. Also, the menu or other GUI displayed on thescreen may be supplanted with specific features adapted for detection,as discussed below.

Next, referring to FIG. 6, at 302, the DSP image processing proceeds toeliminate background image data and isolate the image data for screenimage 202. This processing employs some or all of the above noted uniquecharacteristics of the image 202 to eliminate the background image data.In particular, as shown in FIG. 4 by the shaded area, a majority of thebackground image data 203 will have a brightness substantially less thanimage data 202 and this portion of the background can be rejected byrejecting the pixel data below a reference brightness threshold. Theremaining groups of image data will correspond to relatively brightobjects which may occur in the field of view, illustrated for exemplarypurposes in FIG. 4 by image data 204, 206. For example, such image datamay correspond to a bright object such as a lamp's image data 204. Also,reflected image data 206, for example corresponding to a reflection offof a coffee table or other reflective surface in the field of view maybe present. Image data 204 and 206 may be readily eliminated by usingadditional characteristics of the desired data 202. For example, theundesired image data will in general not have straight edges and not berectangular in shape and therefore may be readily eliminated by thesignal processing. This will employ edge or boundary detection which maybe easily performed since the surrounding pixel data has been eliminatedby the background processing described above and a simple comparison ofpixel values will derive the boundaries of the screen image 202. Also,reflections of the display screen itself may be eliminated by doing acomparison of the brightness of the two images and selecting thebrighter of the two objects. Furthermore, the reflections may besubstantially eliminated from the image data by employing a polarizedfilter in the lens assembly 144. If background image data survives whichsatisfies these characteristics, a comparison between image data 202 andthe surviving image data in the background can be made to determine therelative size of the two objects and the smaller object eliminated.

Additionally, since the image displayed in the display screen image 202is under the control of the input/control device 114 the image 202 maybe provided with a distinct characteristic to enable its ready detectionagainst the background image data in the field of view 200. This displaywill typically correspond to a unique menu or other GUI interface whichcan be provided with a bright boundary, or distinctive bright readilyidentified position markers, e.g. in the four corners of the image 202.This may be superimposed on a standard browser or menu, for example. Onesimple example is illustrated in FIG. 9A which illustrates a displaymenu having a bright background 340. Alternatively a bright boundary 352may be provided or markings 354. Four corner markings 354 are shownwhich may be aligned with image center and which may be relatively smallbright markers. Also these may be combined with boundary 352 with acharacteristic contrast. Also, a distinctive color or combination ofcolors may be employed for boundary 352, markers 354 or both. As anotherexample a logo or name of the device manufacturer (e.g. DVRmanufacturer) or service provider (e.g. cable company) may be used as aunique marker 354. If the logo or name includes text standard OCRprocessing may be used to match the detected combination of letters to astored template for reliable marker detection. Another example of a menulayout is shown in FIG. 9B. It will be appreciated that a variety ofdifferent suitable boundaries and/or markings may be employed to help indistinguishing the image data for the screen 112 from background imagedata. Such characteristics of the display may be combined with orsubstituted for the above noted characteristics for detection of imagedata 202. For example, if discrete markers 354 are employed their uniquecharacteristics may be substituted for the rectangular shape andstraight boundaries of the screen image described above to distinguishfrom background. As another example, if color of the boundary or markeris used, initial color discrimination may be used to reject thebackground.

In the unlikely event that the image processing locks onto an incorrectobject a simple reset may be provided, e.g. using button 140 or someother manually activated input, and upon reset activation a furthercharacteristic may be employed, namely position relative to the centerof the field of view with objects far away from the center beingrejected. This allows the user to reset the image tracking system, forexample if it inadvertently locks onto a window in a room, afterpointing the controller at the display screen and hitting a resetbutton.

After the above noted processing the remaining image data corresponds tothe desired display screen image data 202, as generally illustrated inFIG. 5. The processing flow then proceeds to derive the center of thedisplay screen image from this remaining image data at processing step304, illustrated in FIG. 6. This may employ the rectangular imageboundaries if these have been used at processing 302 or symmetricmarkings 354 if these are used. The process flow next proceeds to derivethe relative position of the center of the screen image 208 to thecenter 210 of the field of view 200 (and the center of the optical axisof the controller lens assembly). As shown in FIG. 5, this offsetinformation may be readily calculated from the image center pixelinformation derived previously and offset values X,Y may be derived asshown. This relative position data is transmitted to the input/controldevice 114 as shown at 306 in FIG. 6. It should be appreciated thatother offset position references may be employed than image center. Forexample, top (and/or bottom) and side boundary offset values may bederived and transmitted to the device 114 as position information.Alternatively, marker position offset(s) from the imager optical axismay be determined and transmitted as position information. Adjustmentfor tilt may be provided during this processing, for example, byrotating the image data about the center of the imager field of viewuntil the edges of the screen or marker correctly align with the edgesof the pixel array before determining offsets. Alternatively, purelyimage feature motion detection may be used for the multi-directionalcontrol, without employing the relative position offset of the imageraxis to the detected image feature. Instead changes in the position ofthe detected image feature between frames may be used to provide motioncontrol. The position information transmitted at 306 may then be justthe change in image position from a prior frame. This approach may alsoprovide correction for tilt, for example, by rotating the image featuredetected at each frame about the imager center to match the prior framebefore determining the change in position. However, while the approachusing imager axis offset information allows either pointing positionbased or motion based control, this approach only allows the latter.

In some applications with sufficiently high data rate transmissionsprovided, some of the above image processing may be performed in device114 rather than in the controller and relatively large amounts of rawimage data transferred. This may reduce the processor size on thecontroller and reduce battery drain. Alternatively, it should also beappreciated that some or all of the processing described below asperformed in device 114 may be performed on board the controller 110.

Next, referring to FIGS. 7 and 8 the control processing using thereceived position data, provided by the input device 114, is shown. Moreparticularly, in FIG. 7 a simplified schematic of the input/controldevice 114 is shown and in FIG. 8 a process flow is illustrated for thetranslation of the position data into cursor control of a GUI interfaceon the display screen 112, shown in FIG. 1. As shown in FIG. 7 the inputdevice 114 will include a receiver 320 for receiving the position dataas well as, optionally, a second receiver 322 for receiving theremote-control input signals from the control inputs 122 on theremote-control. Also, receiver 322 may be in device 116. The receivers320, 322 are coupled to suitable demodulation and amplification circuits324, 326, respectively which in turn provide the received data to amicroprocessor 328. A transmitter 325 and modulator 327 may also beprovided to communicate with the controller 110 or a networked wirelessdevice. Microprocessor 328 will perform a number of functions which willdepend on the particular device and will include additional functionalblocks 330 and 332 for providing control of a GUI interface based onreceived position data from the controller in functional block 330 andoptionally additional remote-control functions from the other inputs 122in block 332. Although these functional blocks are illustrated as partof the system microprocessor 328 it will be appreciated they may be alsoprovided as separate circuits or separately programmed microprocessorsdedicated to the noted functions.

Referring to FIG. 8, a simplified process flow for converting thereceived position data to a multi-directional control function isillustrated. As shown at 350, the process flow begins when a GUI orother multi-directional control mode is entered and the appropriatedisplay will be provided on the display screen 112. As noted above, thisdisplay screen may preferably have a bright background or may includeadditional bright boundaries or other characteristic markings which mayassist in the accurate identification of the screen image data frombackground image data. Two simple examples of such a menu screen areshown in FIGS. 9A and 9B, discussed above. A number of GUI icons 356 arealso illustrated in FIG. 9A along with pointer 118. In FIG. 9B a scrollbar 353 is shown, e.g. for rapid channel selection. If the pointercontrol function is used in web navigation, a bright boundary or markermay be superimposed on the web pages displayed. Next as shown at 360 inFIG. 8 the process flow activated by entry into the multi-directionalcontrol mode operates to receive the position information from thecontroller 110 provided from receiver 320. At 370 the received positioninformation is then processed and translated to cursor positioninformation. Converting the position information to cursor positioncontrol information at 370 may employ a variety of different functionsdepending on the particular application and entertainment systemconfiguration and intended use. In general, this translation operationwill provide a mapping between the received position information andcursor position based on a sensitivity which may be user adjustable. Inparticular, the user may choose to adjust the sensitivity based on howclose the screen is to the user which will affect the amount of angularmotion of the controller 110 required to move the cursor a particularamount in the display screen. Also, the processing at 370 may employ asan input the aspect ratio of the screen and an aspect ratio of thedetected image data 202 may be derived (by the microprocessor 154 or bythe DSP 152 in controller 110 and transmitted along with the positioninformation). These two aspect ratios may be compared to derive an angleat which the user is configured relative to the screen and this anglemay be used to adjust the sensitivity of the received position to cursormap at 370. That is, when the user is directly in front of the screenmovement of the controller will require the maximum angular movement tomove the cursor in the horizontal direction and the control sensitivityof the map of position information to cursor control at 370 may be mademore sensitive. Conversely, when the angle of the user relative to thescreen is greater a smaller movement of the controller will cover therange of cursor movement in the horizontal direction and a lesssensitive mapping at 370 may be employed. In this way the controlfunction will have the same feel irrespective of position of the user.Similarly, compensation processing may be provided at 370 for tilt ofthe controller field of view relative to the screen as well as verticalangle adjustment. Next at 380 the process flow proceeds to compare thechange in cursor positions (and/or change in position data) betweendifferent frames of image data to smooth out the cursor control. Thisprocessing at 380 may be employed to reject jitter by averaging motionover several frames of image data or by rejecting sudden irregularchanges. Such jitter rejection processing may also be adjustable by theuser. Finally at 390 the cursor position (or other position indicated)on the display is updated and the modified GUI screen is displayed.

In another implementation, the cursor itself may be chosen as the uniquemarker displayed on the menu (or other GUI image displayed on the screenby device 114) and its position offset from the imager's center axisdetected. The device 114 will then perform processing to move thedisplayed cursor to eliminate the offset. In other words the cursor willbe moved to where the controller imager axis is pointing. Alternatively,as noted above, instead of controlling a cursor the multi-directionalcontrol may control highlighting of menu items or in a video gameapplication control movement in the video game.

As noted above the use of detected image feature/imager axis positionoffset for motion control of the GUI screen allows either remotepointing position or motion control to be employed. That is the GUIcursor or highlighting control will either be based on where the remoteis pointing on the screen or will simply change based on the change inthe imager axis offset position from frame to frame. The latter controlis of a familiar type to computer users familiar with mouse control andmay be preferred where the primary use of the remote control is in aPC/TV type multi-media system for controlling the PC type functions orin an internet access enabled TV based multi-media system. The pointingposition based control in turn may be preferred for TV menu control,especially for users unfamiliar with mouse control of a PC, or for videogame control. Also, the preferences may change between modes of themulti-media system. For example in a multi-media system with internetaccess, digital TV, and video game capability it may be desirable toswitch modes from pointing position to motion control depending on whataspect of the system is in use. Also, it may be desirable to allow usersto choose their preferred control mode. Therefore, in another aspect ofthe invention one of the buttons illustrated on the remote control 110may be a mode selection button and initiate a corresponding processingmode in the DSP 152 or the control 100 or in the device 114 inmicroprocessor 328 to optionally provide pointing position or motionbased control.

In an embodiment where the controller operates with more conventionaldevices by providing suitable coded high speed left-right-up-downcontrol pulses (e.g. LED pulses) to simulate the use ofleft-right-up-down buttons processing may be provided to convert thedetected position reference information to control pulses transmitted todevice 116 or TV 112. This processing may be performed on controller110. Alternatively, if the universal control is provided via a device114 specially adapted for universal control in conjunction with thecontroller 110 then this processing may be provided in device 114 andthe other device(s) (e.g. 116) controlled via an LED blaster or directconnection such as a USB port.

Referring to FIG. 10, an optional or alternate embodiment of the systemof FIG. 1 is illustrated employing a combined multi-directionalcontroller and wireless keyboard 400. The multi-directional capabilityof controller 400 may be precisely the same as described above inrelation to the controller 110 of the prior embodiments but in additiona keyboard configuration 402 for text entry may be provided. Morespecifically, the controller 400 may have a folding housing with thekeyboard 402 configured in the interior of the controller 400. Thisembodiment may incorporate the teachings of U.S. Pat. No. 6,094,156, thedisclosure of which is incorporated herein by reference in its entirety,and accordingly the details of such an alternate embodiment need not bedescribed in further detail since the necessary modifications tocontroller 110 are apparent from the aforementioned patent. It should beappreciated that such an embodiment may also optionally include inputs122 described above and provide the capabilities of three distinctdevices normally provided by separate controllers, namely a universalremote control, multi-directional control and keyboard text entrycontrol normally found in three separate remote control devices.

Other implementations may be provided that may not be preferred in manyapplications but may be suitable in others. For example, a uniquemarking (such as 354 in FIG. 9) may be provided on the front of display112 or device 114, e.g. by a pattern of LEDs or illuminated distinctivelogo. This could be imaged instead of screen 112 and used to detectcontroller motion for pointer control as above. Alternatively, in suchan embodiment, a distinctive angle sensitive marking may be provided andmotion detected by detection of the changes in brightness of thedetected markings. For example, a light source covered with atransparent diffractive pattern, e.g. a hologram or diffractive gratingmay be used. In yet another embodiment a unique marking may be providedon controller 110 and imaged by device 114 to detect motion. In thisembodiment, the imager 150 is configured in device 114 and the processflow of FIGS. 5-6 is performed in processor 128. All such embodimentsare implied herein.

It will be appreciated that the before mentioned embodiments of theinvention may provide a variety of different capabilities depending onthe particular application and system. In one aspect, the controller 110may be operated as a primary control for a GUI based interface for a TVor other display based entertainment or information system with thefunctionality normally provided by a universal remote provided by theGUI interface and controlled by the multi-directional controller and theadditional controls 122 shown in FIG. 1 need not be provided withoutloss of functionality. Alternatively, the controls 122 may be providedwith the remote controller 110 providing a combined multi-directionalcontroller and universal remote in one simple handheld configuration.Either embodiment may advantageously be used in a TV/cable/DVRenvironment or in a TV/PC enhanced environment and provide enhancedfunctionality. One example of the advantages of such control in adigital cable TV environment is apparent from FIG. 9B. As shown, withscroll bars and selected speed scrolling (by positioning the pointerbelow the listing a desired amount to control scrolling speed) thenavigation of hundreds of channels may be greatly simplified. Also, inanother configuration the multi-directional control capabilities may becombined with a separate wireless keyboard to provide enhancedfunctionality for PC/TV type systems or Internet access systems.Alternatively, the controller may provide multi-directional control andtext entry in the embodiment of FIG. 10. Also, the combination ofuniversal remote, multi-directional controller and text based input mayall be provided in a single compact handheld configuration in theembodiment of FIG. 10. In another application the device 114 may be avideo game device or incorporate video game capabilities and the remotecontrol may provide a control over the video game replacing a joystickor other multi-directional controller.

It will be appreciated by those skilled in the art that the foregoing ismerely an illustration of the present invention in currently preferredimplementations. A wide variety of modifications to the illustratedembodiments are possible while remaining within the scope of the presentconvention. Therefore, the above description should not be viewed aslimiting but merely exemplary in nature.

What is claimed is:
 1. A method comprising: receiving image informationfrom a video camera, the image information capturing a scene whichincludes at least part of a display device, wherein: the video camera isaffixed to a user-manipulated device configured for carrying by a user;and the user-manipulated device comprises a remote control device or acontroller; identifying a plurality of reference fields in the imageinformation to provide identified reference information for controllingan application displayed on the display device, wherein the plurality ofreference fields are displayed as part of graphical informationdisplayed on the screen of the display device, the user-manipulateddevice capable of identifying the plurality of reference fields based oncolors of the plurality of reference fields, based on shapes of theplurality of reference fields, and based on patterns of elements thatcomprise each reference field of the plurality of reference fields;generating position information based on the identified referenceinformation, the position information expressing a position of theuser-manipulated device relative to at least one reference fieldassociated with the display device, wherein the position of theuser-manipulated device is calculated based on a position of the videocamera associated with the user-manipulated device; and controlling theapplication based on the generated position information.
 2. The methodof claim 1, wherein the receiving of the image information comprisesreceiving image information in response to the user pointing at aparticular object being presented by the display device with theuser-manipulated device.
 3. The method of claim 1, wherein the pluralityof reference fields comprise at least two reference fields arrangeddiagonally opposed to one another at opposite corners of the screen ofthe display device.
 4. The method of claim 3, wherein the plurality ofreference fields each include a plurality of reference elements.
 5. Themethod of claim 4, wherein the plurality of reference elements of afirst reference field are arranged in a different pattern from theplurality of reference elements of a second reference field.
 6. Themethod of claim 1, wherein the controlling of the application involvesdetermining an object displayed on the screen of the display device thatthe user is pointing at with the user-manipulated device.
 7. Anapparatus including logic configured to implement the receiving,identifying, generating, and controlling of claim
 1. 8. A systemcomprising: a user-manipulated device, the user-manipulated deviceincluding a video camera associated therewith; a display device forpresenting objects generated by an application; at least one referencefield incorporated as graphical information displayed on a screen of thedisplay device to provide reference information for controlling anapplication displayed on the display device, the user-manipulated devicecapable of identifying the at least one reference field based on one ormore colors of the at least one reference field, based on a shape of theat least one reference field, and based on a pattern of elements thatcomprise the at least one reference field; and a position determinationmodule configured to: receive image information from the video camera,the image information capturing a scene which includes at least part ofthe display device, wherein the user-manipulated device is carried by auser and the video camera moves with the user-manipulated device;identify said at least one reference field in the image information toprovide identified reference information; and generate positioninformation based on the identified reference information, the positioninformation expressing a position of the video camera relative to saidat least one reference field.
 9. The system of claim 8, wherein said atleast one reference field comprises plural reference fields arranged atrespective positions with respect to the display device.
 10. The systemof claim 8, wherein said at least one reference field comprises adefined pattern of plural reference elements.
 11. The system of claim10, wherein: the at least one reference field comprises at least a firstreference field and a second reference field; and the plural referenceelements of a first reference field are arranged in a different patternfrom the plural reference elements of a second reference field.
 12. Thesystem of claim 8, wherein the user-manipulated device is a gamecontroller and the application is a game.
 13. A system comprising: auser-manipulated device, the user-manipulated device including a videocamera associated therewith; a display device for presenting objectsgenerated by an application; a plurality of reference fields placed atdefined positions relative to the display device, each reference fieldcomprising a plurality reference elements arranged in a pattern, whereina first pattern of a first plurality of reference elements differs froma second pattern of a second plurality of reference elements; and aposition determination module configured to: receive image informationfrom the video camera, the image information capturing a scene whichincludes at least part of the display device, wherein theuser-manipulated device is carried by a user and the video camera moveswith the user-manipulated device; identify said at least one referencefield in the image information to provide identified referenceinformation; and generate position information based on the identifiedreference information, the position information expressing a position ofthe video camera relative to said at least one reference field.
 14. Thesystem of claim 13, wherein at least some of the plurality of referenceelements comprise discrete mechanisms coupled to the display device. 15.The system of claim 13, wherein at least some of the plurality ofreference elements comprise graphical information displayed on thedisplay screen.